Interpreting Symptoms of Cognitive Load in Speech Input

Users of computing devices are increasingly likely to be subject to situationally determined distractions that produce exceptionally high cognitive load. The question arises of how a system can automatically interpret symptoms of such cognitive load in the user’s behavior. This paper examines this question with respect to systems that process speech input. First, we synthesize results of previous experimental studies of the ways in which a speaker’s cognitive load is reflected in features of speech. Then we present a conceptualization of these relationships in terms of Bayesian networks. For two examples of such symptoms—sentence fragments and articulation rate—we present results concerning the distribution of the symptoms in realistic assistance dialogs. Finally, using artificial data generated in accordance with the preceding analyses, we examine the ability of a Bayesian network to assess a user’s cognitive load on the basis of limited observations involving these two symptoms. 1 The Challenge for User Modeling When cosmonauts on the space station Mir communicate with ground control, their speech is monitored by psychologists for symptoms of stress (Arnold, 1997). The interpretation of the symptoms in turn influences the nature of the dialogs conducted with the cosmonauts. Computer users do not in general stray quite as far from home as the Mir cosmonauts, nor are they subjected to the same sort of stress. But the mobility of modern computing devices has moved them ever further into the hustle and bustle of everyday life. Situational distractions can have major impact on the quality of interaction with a system—as anyone who has tried to jot down a person’s address with a handheld device while standing on a street corner can testify. For user modeling research, situational distractions represent one more thing that a system can try to recognize and adapt to. Adaptation may involve, for example, a simplification of either the system’s output or the required user input, in cases where situational distractions are suspected. 1.1 Scenario and Field Study For concreteness, consider the example scenario handled by the dialog system READY (see, e.g., Jameson et al., 1999): Users are drivers whose cars need minor repairs; they request assistance from the system in natural language by phone. Our first step in studying this scenario was to get a concrete idea of the cognitive load induced by this situation and the ways in which it is ? This research is being supported by the German Science Foundation (DFG) in its Collaborative Research Center on Resource-Adaptive Cognitive Processes, SFB 378, Project B2, READY. The comments of the two anonymous reviewers strongly influenced the content of the final version. manifested in the users’ speech:1 In a field study conducted on a winter night beside a fairly busy road, each of 8 subjects was given the task of identifying and repairing an intentionally created mechanical problem with a car. They communicated with a professional auto repairman via cellular phone. To get an idea of the information present in features of the subjects’ speech, we analyzed the 8 dialogs in detail: For example, filled and silent pauses were measured and errors were classified. In Sections 2 through 4, we will see how the data from this field study can be analyzed together with results of laboratory experiments of previous researchers so as to yield an empirical basis for a user modeling component for a dialog system. We will then check whether such a user modeling component, if given a sufficiently sound empirical basis, can make usefully accurate inferences on the basis of the limited data about a user that is available in this scenario. 1.2 Determinants of Cognitive Load In this paper, the term cognitive load refers to the demands placed on a person’s working memory by (a) the main task that she is currently performing, (b) any other task(s) she may be performing concurrently, and (c) distracting aspects of the situation in which she finds herself. In the example scenario, we view the main task of the user (U) as that of communicating with the mechanic (or a corresponding system S). Concurrent tasks can involve looking for things, performing actions on the car, or communicating with other persons. Distracting aspects of the situation can include noises and events that interfere with one’s concentration on task performance, as well as internal factors like emotional stress that have similar effects. In the dynamic Bayesian networks that form the core of READY’s user model, these types of influence on a user’s continually changing cognitive load are modeled separately (see, e.g., Jameson et al., 1999; Schäfer and Weyrath, 1997). In this paper, we will simply consider the problem of assessing the total load currently placed on U’s working memory, regardless of its origin. This load will be assumed to remain constant throughout the period during which it is being assessed. 2 Overview of Symptoms and Their Modeling We reviewed literature from psycholinguistics and linguistics looking for evidence concerning the effects of cognitive load on features of speech. Table 1 gives a high-level summary of the results of this survey.2 Figure 1 shows how the relationships between these symptoms and cognitive load can be modeled with a Bayesian network.3 To see the meaning of the variables, suppose that various factors have created a POTENTIAL WM LOAD for U . If this load is too great for U to handle without 1 The READY system also tries to recognize and adapt to the user’s time pressure. For reasons of space, this variable will be mentioned only in passing in this paper. 2 A much more detailed discussion of these results is given by Berthold (1998), along with references to the individual studies and results for less important features not listed here. 3 For introductions to Bayesian networks, see, e.g., Russell and Norvig (1995) or Pearl (1988). An overview of their applications to user modeling is given by Jameson (1996). Table 1. Summary of previous results concerning potential speech symptoms of cognitive load. Symptoms involving output quality Symptoms involving output rate Feature Tendency Tally Feature Tendency Tally Sentence fragments (number) + 4/5 Articulation rate 7/7 False starts (number) + 2/4 Speech rate 7/7 Syntax errors (number) + 1/1 Onset latency (duration) + 9/11 Self-repairs (number) +, , 0 2, 1, 4 Silent pauses (number) + 4/5 Silent pauses (duration) + 8/10 Filled pauses (number) + 4/6 Filled pauses (duration) + 1/2 Repetitions (number) + 5/6 a “+” means that the measure was generally found to increase under conditions of high cognitive load; “ ” means the opposite. b “m/n” means that of n relevant studies,m found the tendency indicated in the second column. (In most— but not all—cases the tendency was statistically significant.) c Results concerning self-repairs show an inconsistent pattern. difficulty, U may cope with the overload by reducing the speed of speech generation—for example, by pausing intermittently to think or to deal with distractions. (The extent to which U does this can be influenced by features of the task as well as by U’s time pressure and preferences.) Any such speed reduction can be reflected in specific symptoms like the ones shown on the right in Table 1. Because of the slowing, the ACTUAL WM LOAD—which can be conceptualized as the amount of cognitive work that has to be done in a given unit of time—will be reduced. On the other hand, U may for various reasons avoid slowing down, or may slow down only to a degree that is inadequate to reduce the ACTUAL WM LOAD to a normal level. In this case, the high ACTUAL WM LOAD is likely to be reflected in various types of defect in the utterances produced, such as the types listed in the left-hand side of Table 1 (cf. the left-hand side of Figure 1).4 So far, we are aware of only partial and indirect evidence in favor of the speed-accuracy tradeoff postulated in Figure 1. Concerning the relationships between the nodes for the individual symptoms and their parent nodes, useful empirical data can be extracted from the studies summarized in Table 1 and from our own field study that was sketched above. The next two sections will show how this can be done, using one example from each of the two broad categories of symptoms, starting with one that involves a decline in the quality of output. 3 Sentence Fragments as a Symptom A sentence fragment can be defined as an incomplete syntactic structure I for which there exists a syntactic continuation C such that I C constitutes a well-formed sentence. After articulating I, 4 Baber et al. (1996), while not explicitly postulating the relationships depicted in Figure 1, discuss a number of phenomena and relationships that are consistent with this account.